Location-Specific Three-Dimensional Models Responsive to Location-Related Queries

ABSTRACT

Generating a location-specific three-dimensional model in response to a location query can provide users with a better understanding of a location through providing better interactivity, better perspective, and better understanding of dimensionality. Generation of the models can be enabled by leveraging a three-dimensional asset database and segmentation methods. The location-specific models can provide further utility by further including situation specific simulated effects, such as simulated weather or traffic.

FIELD

The present disclosure relates generally to providing athree-dimensional model of a location in response to a user query forinformation that is relevant or related to a location. Moreparticularly, the present disclosure relates to providing alocation-specific segment of a three-dimensional model of a location inresponse to a location-related query for information associated withthat location such as, for example, a search query, a navigationalquery, a mapping query, a weather query, a review query, a visual query,and/or other user requests for location-related information.

BACKGROUND

When searching for locations via a search engine, certain existingsystems can return results in the form of hyperlinks and/or a generatedgraphic that can include a textual snippet, a photograph, or a map. Formonuments and landmarks, these results often fail to depict the actuallook and magnitude of the monument or landmark. Moreover, images canprovide a perspective of what the location may look like, but imagesalone can fail to capture the dimensionality of the location.Furthermore, videos of locations can also be limited for a variety ofreasons. In particular, the search results lack interactivity, which canmake finding different aspects of the location difficult (e.g., contexton the location or different perspective views of the location). Forexample, trying to find close-ups of different perspectives or lookingat different portions of the location can require tedious searchinginstead of being provided in an intuitive, interactive interface.

Knowing the weather, traffic, and crowd size at monuments and landmarkscan help tourists and other visitors better understand and prepare for avisit to the monument. Maps alone fail to put traffic or crowd size intoperspective. Moreover, monument or landmark visibility can be affectedby crowd size or weather. Knowledge about the effect of these factors tovisibility can often be limited to scouring literature on the subject.However, many people consider themselves visual learners.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to acomputer-implemented method for supplying three-dimensional modeling toa user. The method can include obtaining, by a computing system that caninclude one or more computing devices, a location query associated witha user computing device. The method can include processing, by thecomputing system, the location query to determine a location. In someimplementations, the method can include accessing, by the computingsystem, a three-dimensional asset database to retrieve athree-dimensional model that models the location in combination with oneor more other, different locations. The method can include segmenting,by the computing system, the three-dimensional model to generate alocation-specific segment of the three-dimensional model thatcorresponds to the location. In some implementations, thelocation-specific segment can model the location in isolation from theone or more other, different locations. The method can includeproviding, by the computing system, the location-specific segment of thethree-dimensional model or one or more renderings of thelocation-specific segment of the three-dimensional model to the usercomputing device.

Another example aspect of the present disclosure is directed to acomputing system. The computing system can include one or moreprocessors and one or more non-transitory computer readable media thatcollectively store instructions that, when executed by the one or moreprocessors, cause the computing system to perform operations. Theoperations can include obtaining a location query from a user computingdevice. The operations can include processing the location query todetermine a location. In some implementations, the operations caninclude accessing a three-dimensional asset database to retrieve athree-dimensional model associated with the location. The operations caninclude generating a location-specific model by segmenting thethree-dimensional model to isolate the location from other objects inthe three-dimensional model and providing the location-specific model tothe user computing device.

Another example aspect of the present disclosure is directed to one ormore non-transitory computer readable media that collectively storeinstructions that, when executed by one or more processors, cause acomputing system to perform operations. The operations can includeobtaining a user input that specifies a location query that relates to alocation. The operations can include providing the location query to aserver system over a network. In some implementations, the operationscan include receiving, from the server system over the network, alocation-specific segment of a three-dimensional model or one or morerenderings of the location-specific segment of a three-dimensionalmodel. The three-dimensional model can model the location in combinationwith one or more other, different locations, and the location-specificsegment of a three-dimensional model can be generated by segmenting thethree-dimensional model to isolate the location from the one or moreother, different locations. The operations can include providing fordisplay at least a first rendering of the location-specific segment ofthe three-dimensional model.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, user interfaces,and electronic devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example computing system thatperforms a location-based model search according to example embodimentsof the present disclosure.

FIG. 2 depicts a block diagram of an example model search systemaccording to example embodiments of the present disclosure.

FIG. 3 depicts a block diagram of an example situation-aware modelsearch system according to example embodiments of the presentdisclosure.

FIG. 4 depicts a block diagram of an example three-dimensional renderingaccording to example embodiments of the present disclosure.

FIG. 5 depicts an illustration of an example search results pageaccording to example embodiments of the present disclosure.

FIG. 6 depicts a flow chart diagram of an example method to performlocation-based model search according to example embodiments of thepresent disclosure.

FIG. 7 depicts a flow chart diagram of an example method to performlocation-based model search according to example embodiments of thepresent disclosure.

FIG. 8 depicts a flow chart diagram of an example method to performthree-dimensional model database creation according to exampleembodiments of the present disclosure.

FIG. 9A depicts an illustration of an example three-dimensional modelaccording to example embodiments of the present disclosure.

FIG. 9B depicts an illustration of an example situation-awarethree-dimensional model according to example embodiments of the presentdisclosure.

FIG. 10 depicts an illustration of an example augmented-realityexperience according to example embodiments of the present disclosure.

FIG. 11 depicts an illustration of an example rendering displayaccording to example embodiments of the present disclosure.

FIG. 12 depicts an illustration of an example route overlay according toexample embodiments of the present disclosure.

Reference numerals that are repeated across plural figures are intendedto identify the same features in various implementations.

DETAILED DESCRIPTION Overview

Generally, the present disclosure is directed to providing alocation-specific segment of a three-dimensional model of a location inresponse to a location-related query for information associated withthat location such as, for example, a search query, a navigationalquery, a mapping query, a weather query, a review query, a visual query,and/or other user requests for information. The systems and methodsdisclosed herein can leverage a database of three-dimensional models forproviding location-based renderings and/or location-specific models inresponse to a location-related query (a “location query”). In someimplementations, the systems and methods can include obtaining alocation query. The location query can be obtained from a user computingdevice or may be triggered by a user computing device. The systems andmethods can include processing the location query to determine alocation. In some implementations, the systems and methods can includeaccessing a three-dimensional asset database to retrieve athree-dimensional model associated with the location. Thethree-dimensional model and the location can be used to generate alocation-specific model. Generating the location-specific model caninvolve segmenting the three-dimensional model to isolate the locationfrom other objects in the three-dimensional model. The systems andmethods can include providing the location-specific model to a computingsystem. For example, the location-specific model may be provided to theuser computing device. In some implementations, the location-specificmodel can include an architectural structure (e.g., a landmark, amonument, and/or a building).

Obtaining the location query can be facilitated by a user interface thatis configured to receive input from a user. Alternatively and/oradditionally, the location query may be a query generated by andobtained by the computing system in response to a user computing devicetriggered event. For example, a user may be scrolling through a list ofattractions and selects a particular attraction to learn more about theattraction. In response to the selection, a location query for theattraction may be generated and obtained for processing. In otherexamples, a user can request to view the location on a map, requestnavigational instructions to the location, request information (e.g.,user-generated reviews) about a point of interest associated with thelocation, submit a textual or visual search query for which the locationis a result, and/or other actions which indicate a request forinformation related to the location. Each of these example actions cantrigger a location query for processing by the proposed systems.

In some implementations, the location query can include a string ofterms or one or more images that are associated with a particularlocation. The terms may describe the location, may include the name ofthe location, may be the address of the location, or may be any otherform of indirect or directly related information that implicates thelocation or for which the location is a relevant result. In someexamples, the location query may include images for processing anddetermining where the picture was taken or what is depicted in theimage.

Once a location is determined, a three-dimensional asset database may beaccessed and searched to determine if the particular location has anyassociated three-dimensional models. The three-dimensional model can bea constructed three-dimensional model of a particular structure or aplurality of structures that are found at that location. In particular,in some implementations, the three-dimensional asset database cancontain a general or global three-dimensional model that spans asignificant portion of the surface of the Earth or of individualcountries or cities located therein. For example, various algorithms canbe applied to automatically generate from available imagery (e.g.,satellite imagery and/or street-level imagery) one or more very largethree-dimensional models that span various addresses, streets,neighborhoods, cities, and/or the like.

To provide a location-specific model, the systems and methods canisolate a location in the general three-dimensional model to generate alocation-specific model. Specifically, example systems of the presentdisclosure can segment portions of the general three-dimensional modelinto multiple segments. In some implementations, the systems and methodscan perform three-dimensional segmentation to extract alocation-specific segment of the three-dimensional model that models oneor more specific locations. The location-specific model may be generatedusing image segmentation.

After the location-specific segment of the model is generated, thelocation-specific segment of the three-dimensional model can be renderedand displayed to the user in various contexts. As one example, a servercomputing system can transmit or serve the location-specific segment ofthe three-dimensional model to the user computing device, and the usercomputing device can render a rendering of the location using thelocation-specific model. Alternatively and/or additionally, the servercomputing system can perform rendering of the location-specific segmentof the three-dimensional model and can transmit or serve the renderingto the user computing device.

In some implementations, the location-specific segment of thethree-dimensional model can be generated by image segmentation using aplurality of images associated with the location. Moreover, theplurality of images can include one or more aerial images. Alternativelyand/or additionally, in some implementations, the model may be generatedusing Light Detection and Ranging (LiDAR) data or other sensor data(e.g., spectrum sensor data generated using spectrum sensors, which canbe used to segment trees or other objects). In particular, the LiDARdata or other sensor data can be used to determine what should besegmented.

The location-specific segment of the three-dimensional model may beprovided in various methods via various mediums. In someimplementations, the location-specific model can be provided as anaugmented-reality asset for rendering in a user's environment via anaugmented-reality experience. Alternatively and/or additionally, thelocation-specific model can be provided in a virtual-reality experience.In some implementations, the user computing device can be provided withan interactive virtual environment that can include thelocation-specific model. Moreover, the interactive environment caninclude one or more two- or three-dimensional maps. Thelocation-specific segment of the three-dimensional model can be providedin a pre-rendered video or a set of pre-rendered videos, in which eachvideo may depict different views of the location generated based on thelocation-specific segment.

In some implementations, the location-specific model may be providedwith other information or data related to or associated with a location.For example, the systems and methods can include retrieving an addressof the location and providing the address of the location with thelocation-specific model. In some implementations, the systems andmethods can include retrieving location information data. The locationinformation data can include information pertaining to the location(e.g., information on the history of the location). The locationinformation data can be provided with the location-specific model to theuser computing device. In some implementations, the location-specificmodel may be provided with a plurality of images of the location.Moreover, in some implementations, the systems and methods can includegenerating a map that includes the location-specific model and providingthe map to the user computing device. Additionally, in someimplementations, the systems and methods can include obtaining locationdata for the user computing device. The location data can be descriptiveof a user location. The systems and methods can include obtaining anaddress for the location associated with the location-specific model.The address and the user location can be used to determine a route for auser to take to get from the user location to the address. A visualroute overlay may be generated based on the route, and the visual routeoverlay and the map may be used to generate an augmented map with thevisual route overlay being superimposed on the map. The augmented mapcan then be provided to a user computing device.

Alternatively and/or additionally, the systems and methods can includeobtaining situation data. The situation data can be descriptive of atime-specific situation associated with the location. The systems andmethods can include processing the situation data to determine arespective simulation asset and retrieving the respective simulationasset. The respective simulation asset and the location-specific modelmay be used to generate a simulated event that includes the timespecific situation occurring at the location. In some implementations,the situation may be a weather condition, locations of various physicalitems such as trains, airplanes, buses, taxis, and/or the like, a crowdsize, and/or traffic at or around the location. For example, thetime-specific situation may be rain, and the generated simulated eventmay be a rendering of the location-specific segment of thethree-dimensional model with simulated rain.

In some implementations, the providing of the location-specific modelmay include providing the location-specific model with one or moresearch results via a webpage (e.g., a webpage returned in response to atextual or visual location query). Alternatively and/or additionally,the location-specific model may be provided and displayed in a separatesection or location on the webpage than the other search results. Forexample, in some implementations, the location-specific model may beprovided in a separate pane with other data on the location associatedwith the model (e.g., the name of the location associated with thelocation-specific model, address of the location, and/or information onthe location). In some implementations, the mesh segmentation can allowthe responses to queries to be compact. Specifically, a segment of amesh can be rendered (e.g., in a pane) more quickly than a rendering ofthe whole region. The segmentation rendering can therefore aid inproviding search results without having the heavy taxation on bandwidththat full model rendering can cause.

The systems and methods disclosed herein can be utilized in a variety offields and industries. For example, the systems and methods disclosedherein can be utilized to provide an enhanced user experience for a mapapplication. More specifically, the systems and methods disclosed hereincan be used to provide a map to a user with a location-specific model inresponse to a location query. The map application may provide a visualroute overlay to provide a realistic rendering of the area attempting tobe navigated through. In some implementations, the map application canprovide several options for understanding an area. The map applicationcan provide a street view of the location, an overhead map, an aerialview map, or an interactive three-dimensional model.

Another example can include implementing the systems and methods intovideo games and/or augmented or virtual reality experiences. Forexample, the systems and methods disclosed herein can allow for thegeneration of an open world game in which a user can travel to differentareas and upon reaching an area, a location query may be sent, and alocation-specific model of the area may be received in response. Morespecifically, the game can be set in France, and when a charactertravels to a location, a location query for that location can be sent toa server and a location-specific model of the location may be receivedin return (e.g., the character may travel to the Eiffel Tower, alocation query for the Eiffel Tower may be sent, and a location-specificmodel of the Eiffel Tower may be received in return.).

The three-dimensional asset database leveraged by the systems andmethods disclosed herein may be generated with a variety of methods orprocesses. The processes can begin with the creation of thethree-dimensional content. The three-dimensional environment can bebuilt through general three-dimensional reconstruction methods. Thereconstruction can generate three-dimensional meshes with attachedtextures (e.g., two-dimensional images that represent the color of themesh). The next step of the process can involve image segmentation.Segmentation can be used to understand the meshes and textures to findthe underlying semantic structures of the environment (e.g., thesemantic structure of a reconstructed city). In some implementations,the process can utilize semantic manifolds for image segmentation. Theimage segmentation can involve segmenting a plurality of two-dimensionalimages into parts with a machine-learned model. The segmented images canthen be projected into the three-dimensional meshes. The process canfurther involve cleaning up the generated three-dimensional environmentby removing blobs and cleaning up the mesh. For example, trees, streets,sidewalks, and other objects may be removed. The cleaning up step can becompleted manually or can be automated. The resulting three-dimensionalasset can be compressed and can be stored in a database. The process canbe repeated iteratively to update the database with up to datestructures from a city, town, etc.

The systems and methods of the present disclosure provide a number oftechnical effects and benefits. As one example, the system and methodscan provide a location-specific segment of a three-dimensional model inresponse to a location query. The systems and methods can further beused to provide a model of the location (e.g., a landmark or monument)with real-time traffic, crowd size, weather, and/or other contextual orsituation-specific information. Furthermore, the systems and methods canenable three-dimensional models for map applications.

Another technical benefit of the systems and methods of the presentdisclosure is the dimensionally accurate modeling and the interactivityof the provided rendering. The model can provide dimensionally accuraterepresentations of locations to allow for potential visitors todetermine if they indeed wish to visit the location.

Another example technical effect and benefit relates to improvedcomputational efficiency and improvements in the functioning of acomputing system. For example, certain existing systems allow a user toview renderings of a global or general three-dimensional model.Processing and rendering the entire global or general three-dimensionalmodel requires significant computational resources. In contrast, byextracting and then rendering a location-specific segment of thethree-dimensional model, the proposed systems and methods can savecomputational resources such as processor usage, memory usage, and/ornetwork bandwidth.

With reference now to the Figures, example embodiments of the presentdisclosure will be discussed in further detail.

Example Devices and Systems

FIG. 1 depicts an exemplary computing system 100 that can be used toimplement a location-based model search according to aspects of thepresent disclosure. The system 100 has a client-server architecture thatincludes a server 110 that communicates with one or more client devices130 over a network 160. However, the present disclosure can beimplemented using other suitable architectures, such as a singlecomputing device unconnected to a network.

The system 100 includes a server 110, such as, for example, a webserver. The server 110 can be one or more computing devices that areimplemented as a parallel or distributed computing system. Inparticular, multiple computing devices can act together as a singleserver 110. The server 110 can have one or more processor(s) 112 and amemory 114. The server 110 can also include a network interface used tocommunicate with one or more remote computing devices (e.g., clientdevices) 130 over a network 160.

The processor(s) 112 can be any suitable processing device, such as amicroprocessor, microcontroller, integrated circuit, or other suitableprocessing device. The memory 114 can include any suitable computingsystem or media, including, but not limited to, non-transitorycomputer-readable media, RAM, ROM, hard drives, flash drives, or othermemory devices. The memory 114 can store information accessible byprocessor(s) 112, including instructions 116 that can be executed byprocessor(s) 112. The instructions 116 can be any set of instructionsthat when executed by the processor(s) 112, cause the processor(s) 112to provide desired functionality.

In particular, the instructions 116 can be executed by the processor(s)112 to implement a three-dimensional model search 120. Thethree-dimensional model search 120 can be configured to search for athree-dimensional model associated with a location to segment and outputa location-specific three-dimensional model. In some implementations,the three-dimensional model search 120 can be configured to access athree-dimensional asset database 180 to retrieve the three-dimensionalmodel, and the three-dimensional model can be segmented using the imagesegmentation element 122 stored in the memory 114 of the servercomputing system 110.

It will be appreciated that the term “element” can refer to computerlogic utilized to provide desired functionality. Thus, any element,function, and/or instructions can be implemented in hardware,application specific circuits, firmware and/or software controlling ageneral purpose processor. In one implementation, the elements orfunctions are program code files stored on the storage device, loadedinto memory and executed by a processor or can be provided from computerprogram products, for example computer executable instructions, that arestored in a tangible computer-readable storage medium such as RAM, harddisk or optical or magnetic media.

Memory 114 can also include geographic data 118 that can be retrieved,manipulated, created, or stored by processor(s) 112. Geographic data 118can include geographic imagery (e.g., digital maps, satellite images,aerial photographs, street-level photographs, synthetic models, etc.),tables, vector data (e.g., vector representations of roads, parcels,buildings, etc.), point of interest data (e.g., locales such as islands,cities, restaurants, hospitals, parks, hotels, and schools), or othersuitable geospatial data or related information. As an example,geographic data 118 can be used to access information and dataassociated with a location and generate a rendering of athree-dimensional model of the surface of the Earth.

The geographic data 118 can be stored in one or more databases. The oneor more databases can be connected to the server 110 by a high bandwidthLAN or WAN, or can also be connected to server 110 through network 160.The one or more databases can be split up so that they are located inmultiple locales.

The server 110 can exchange data with one or more client devices 130over the network 160. Although two clients 130 are illustrated in FIG.1, any number of client devices 130 can be connected to the server 110over the network 160. The client devices 130 can be any suitable type ofcomputing device, such as a general purpose computer, special purposecomputer, navigational device, laptop, desktop, integrated circuit,mobile device, smartphone, tablet, wearable-computing devices, a displaywith one or more processors coupled thereto and/or embedded therein, orother suitable computing device. Further, client device 130 can bemultiple computing devices acting together to perform operations orcomputing actions.

Similar to server 110, a client device 130 can include a processor(s)132 and a memory 134. The memory 134 can store information accessible byprocessor(s) 132, including instructions that can be executed byprocessor(s) and data. As an example, memory 134 can store a browserelement 140 and an application element 142.

Browser element 140 can provide instructions for implementing a browser.In particular, the user of client device 130 can exchange data withserver 110 by using the browser to visit a website accessible at aparticular web-address. The three-dimensional model search of thepresent disclosure can be provided as an element of a user interface ofthe website.

Application element 142 can provide instructions for running aspecialized application on client device 130. In particular, thespecialized application can be used to exchange data with server 110over the network 160. Application element 142 can includeclient-device-readable code for providing and implementing aspects ofthe present disclosure. For example, application element 142 can provideinstructions for implementing a mapping application or a virtual globeapplication.

The client device 130 can include various user input devices 150 forreceiving information from a user, such as a touch screen, touch pad,data entry keys, speakers, mouse, motion sensor, and/or a microphonesuitable for voice recognition. Further, the client device 130 can havea display 146 for presenting information, such as a rendering of athree-dimensional model in a three-dimensional environment.

The client device 130 can also include a positioning system 148 that canbe used to identify the position of the client device 130. Thepositioning system 148 can be optionally used by the user to monitor theuser's position relative to the rendering. The positioning system 148can be any device or circuitry for monitoring the position of the clientdevice 130. For example, the positioning device 148 can determine actualor relative position by using a satellite navigation positioning system(e.g., a GPS system, a Galileo positioning system, the GLObal Navigationsatellite system (GLONASS), the BeiDou Satellite Navigation andPositioning system), an inertial navigation system, a dead reckoningsystem, based on IP address, by using triangulation and/or proximity tocellular towers or WiFi hotspots, and/or other suitable techniques fordetermining position.

The client device 130 can further include a graphics processing unit152. Graphics processing unit 152 can be used by processor 132 to renderor depict three-dimensional imagery. In some embodiments, client device130 performs any and all three-dimensional rendering required forproviding the three-dimensional environment.

The client device 130 can include a network interface 154 forcommunicating with server 110 over network 160. Network interface 154can include any components or configuration suitable for communicationwith server 110 over network 160, including, for example, one or moreports, transmitters, wireless cards, controllers, physical layercomponents, or other items for communication according to any currentlyknown or future developed communications protocol or technology.

The network 160 can be any type of communications network, such as alocal area network (e.g., intranet), wide area network (e.g., Internet),or some combination thereof. The network 160 can also include a directconnection between a client device 130 and the server 110. In general,communication between the server 110 and a client device 130 can becarried via network interface using any type of wired and/or wirelessconnection, using a variety of communication protocols (e.g., TCP/IP,HTTP), encodings or formats (e.g., HTML, XML), and/or protection schemes(e.g., VPN, secure HTTP, SSL).

Example Model Arrangements

FIG. 2 depicts a block diagram of an example model search system 200according to example embodiments of the present disclosure. In someimplementations, the model search system 200 is trained to receive a setof input data 202 descriptive of a search query and, as a result ofreceipt of the input data 202, provide output data 206 that includes oneor more search results. Thus, in some implementations, the model searchsystem 200 can include a search engine 204 that is operable to process asearch query and determine intent.

The example model search system 200 can involve a search engine 204obtaining a search query 202 as input and outputting search results 206which can include one or more location-specific models. The search query202 can be a location search query associated with a location. Thesearch engine 204 can process the query to determine the locationassociated with the search query 202. The search engine can then accessa location database 208 to retrieve data related to the location and mayaccess a three-dimensional asset database 210 to determine if thelocation has a related three-dimensional model. If a three-dimensionalmodel is found, the three-dimensional model may be segmented to generatea location-specific model, which can be output as a search result 206.In some implementations, the search results 206 can further include oneor more links based on the search query 202 and may include locationdata retrieved from a location database 208.

FIG. 3 depicts a block diagram of an example situation-aware modelsearch system 300 according to example embodiments of the presentdisclosure. The situation-aware model search system 300 is similar tothe model search system 200 of FIG. 2 except that situation-aware modelsearch system 300 further includes situation aware search and rendering.

The example situation-aware model search system 300 can involve a searchengine 304 obtaining a search query 302 as input and outputting searchresults 306 which can include one or more location-specific models. Thesearch query 302 can be a location search query associated with alocation. The search engine 304 can process the query to determine thelocation associated with the search query 302. The search engine canthen access a location database 308 to retrieve data related to thelocation and may access a three-dimensional asset database 310 todetermine if the location has a related three-dimensional model. If athree-dimensional model is found, the three-dimensional model may besegmented to isolate the location to generate a location-specific model,which can be output as a search result 306. In some implementations, thesearch results 306 can further include one or more links based on thesearch query 302 and may include location data retrieved from a locationdatabase 308. In some implementations, the search engine can furtheraccess time specific situation data related to the location usingsituation query processing 312, which can be used to determine the timespecific situation. Once a time specific situation is determined, asimulation asset database 314 can be accessed to retrieve a simulationrendering effect descriptive of the determined situation. Thelocation-specific model and the simulation rendering effect can be usedto generate a simulated event to be output as a search result. Thesimulated event can include a three-dimensional model of a location withthe situation rendered into the model. For example, the search query maybe related to the Washington Monument, and the time specific situationmay include the current weather state, which includes rain. Therefore,the simulated event can be a model of the Washington Monument with rainrendered into the model.

FIG. 4 depicts a block diagram of an example results page 400 accordingto example embodiments of the present disclosure. In someimplementations, the results page 400 can include a search query inputbar 402 and, as a result of inputting a search query into the searchquery input bar 402, provide one or more search results 404 that caninclude links, text, and/or media in response to the search query input.In some implementations, the results page 400 can include a knowledgegraph 410 that is operable to provide a group of data related to one ormore search results.

The example results page 400 of FIG. 4 can be output in response to asearch query. The search query can be a location search query and can bea search query input into a web search service. The search results page400 can include a search query input bar 402 for inputting new queriesor refining the current query. In some implementations, the search queryinput bar 402 can continue to display the last search query input or canbe emptied upon retrieving the search results. Moreover, the searchresults page 400 can further include one or more search results 404. Thesearch results 404 can include links, images, snippets, maps, videos,and/or documents related to a determined search intent. The searchresults page 400 can include a knowledge graph 410, which can includeone or more media items 412, a location-specific model 414, a locationname 416, and location information 418. The knowledge graph 410 can begenerated based on a predefined function, one or more machine-learnedmodels, and/or a pre-defined knowledge graph associated with thelocation. The one or more media items 412 can be images, videos, graphs,maps, or audio data related to the location. Moreover, thelocation-specific model 414 may be generated by retrieving andsegmenting a three-dimensional model in response to retrieving thesearch query. The location name 416 may be a formal name or a translatedname of the location determined from the search query. Lastly, thelocation information 418 can include a variety of information onpopulation, history, architecture, or size of the location. The locationinformation 418 can be extracted from one or more search results or canbe data paired with the location name in a location database.

FIG. 5 depicts an illustration of an example three-dimensional rendering500 according to example embodiments of the present disclosure. In someimplementations, the three-dimensional rendering 500 can providedifferent perspective views, that can include an aerial view.

The example three-dimensional rendering 500 of FIG. 5 displays oneperspective of a location. The rendering 500 can be displayed inresponse to a search query and may be rendered based on a providedlocation-specific model. The location-specific model can be provided inan interactive environment to provide a user with the ability tonavigate the location-specific model to view different respectiveperspectives of the location, including an aerial view 502 or variousside views 504 and 506. The rendering of the different perspectives canbe based on the location-specific model and may occur at either theserver computing system or the client computing system. Alternativelyand/or additionally, the location specific-model can be provided as partof a pre-rendered media clip in which the three-dimensional rendering500 and the different perspective views 502, 504, and 506 can be part ofa video, slide show, or other display medium. In some implementations,the location-specific model may be provided as part of a virtual-realityexperience, such that the three-dimensional rendering 500 can be one ofa plurality of renderings viewed by a user while navigating the virtualspace. Generation of the location-specific three-dimensional model canutilize geometry extraction and may use Unreal Engine to display themodel.

FIG. 9A depicts an illustration of an example three-dimensionalrendering 900 according to example embodiments of the presentdisclosure.

A three-dimensional rendering 900 can be based on a three-dimensionalmodel and may be a rendering of any perspective, which can include oneor more aerial perspectives. The three-dimensional rendering 900 of FIG.9 is based on a location-specific model related to the Space Needle inSeattle. The rendering 900 includes one perspective, but alternativerenderings can be generated based on the location-specific model,including but not limited to an aerial view 902 or a plurality of sideviews 904 and 906. In some implementations, the rendering 900 caninclude situation specific renderings to provide situation specificsimulations. For example, this specific rendering 900 displays the SpaceNeedle during an overcast day. However, Seattle is not alwaysprecipitation free and alternatively is not always overcast. A user maywant a simulation of what the Space Needle looks like during variousweather states. In some implementations, the location-specific model canbe provided to a user computing device without outside features orobjects. The location-specific model could then be viewed completelyisolated or outside environmental features may be added by the usercomputing device based on user input.

FIG. 9B depicts illustrations of example three-dimensional renderings910, 920, and 930 according to example embodiments of the presentdisclosure. The three-dimensional renderings 910, 920, and 930 aresimilar to the three-dimensional rendering 900 of FIG. 9A except thatthe three-dimensional renderings 910, 920, and 930 further includesituation based simulations.

The systems and methods disclosed herein can be used to retrieve orgenerate situation data, which can be used to retrieve simulation assetsrelated to the situation described by the situation data. The simulationassets can be used to generate a simulated event with thelocation-specific model, which can include a simulation of the specificsituation. For example, FIG. 9B displays three simulated weather statesgenerated using the location-specific model and a simulation asset. Thethree-dimensional renderings are displayed in a user-interface with alocation and the weather state. A partly cloudy three-dimensionalrendering 910 can provide a simulated partly cloudy weather state forthe location, which can provide a user with a rendering that accuratelydepicts that location with that particular weather state. Moreover, thesystems and methods disclosed herein can provide previews of thelocation in various other weather states such as a rainy weather state920 or a heavy rain weather state 930. These renderings can be timespecific to an input time by a user, can be based on a real-timesituational state, or can be a manually selected situational state.Alternatively and/or additionally, the situation can be a crowdsituation, a traffic situation, and/or a variety of other situationsthat may be determined and simulated for the user.

FIG. 10 depicts an illustration of an example augmented-realityexperience 1020 according to example embodiments of the presentdisclosure. In some implementations, the augmented-reality experience1020 is trained to receive a set of input data descriptive of athree-dimensional model and, as a result of receipt of the input data,provide output data 1024 that can be a rendering of a three-dimensionalmodel of a location. Thus, in some implementations, theaugmented-reality experience 1020 can include a preview of a location1012 that is operable to depict the location before an augmented-realityexperience is selected 1016.

The augmented-reality experience 1020 can be one of many mediums inwhich to provide the location-specific model. In this implementation,the user computing device is provided with a user interface that allowsthe user to view the location-specific model 1012 in context 1010 or asan augmented-reality rendering 1024. In this implementation, the userinterface provides the location-specific model in context 1010 with thelocation name and a textual snippet 1014 with information on thelocation. The user interface can contain a selectable icon 1016 toswitch to an augmented-reality experience 1020 that can use thelocation-specific model 1012 to render the location into a user'senvironment 1022. The augmented-reality rendering 1024 can be placed inthe user's environment 1022 via the augmented-reality experience 1024 toview the location as a scaled down model. Moreover, the user interfacecan provide a selectable icon 1026 to exit the augmented-realityexperience 1020 and return to the location-specific model 1012 incontext 1010.

FIG. 11 depicts an illustration of an example media results page 1100according to example embodiments of the present disclosure. In someimplementations, the media results page 1100 can include a set of imagesof a location 1104, 1106, and 1108, and a location-specificthree-dimensional model 1110. Furthermore, in some implementations, themedia results page 1100 can include an interactive environment that isoperable to view and interact with the location-specificthree-dimensional model 1110.

The media results page 1100 in FIG. 11 includes a plurality of imageresults 1104, 1106, and 1108, the search query 1102, a three-dimensionalmodel of the location, a map of the location, and the name of thelocation 1112. In the depicted implementation, the search query 1102includes the name of the location, but in other implementations, thesearch query can include an address, descriptors of the location, or anyother query that can be associated with the location. The plurality ofimages 1104, 1106, and 1108 can be displayed as a side panel withselectable tabs at the top of the panel to refine the results or depictdifferent forms of media. Moreover, the media results page 1100 canprovide links to source data or other pages by selecting the media orvarious selectable icons.

The location-specific three-dimensional model 1110 can be provided in amain pane, a solo pane, or on the side pane. The location-specific model1110 can be provided in an interactive environment or in a pre-renderedmedia form. In some implementations, the media results page 1100 caninclude location information or location-specific data superimposed overone or more of the media items. For example, in this implementation, amap of the location and its surrounding area, the location name 1112,and information on the location are superimposed over thethree-dimensional model 1110.

FIG. 12 depicts an illustration of an example route overlay model 1200according to example embodiments of the present disclosure. In someimplementations, the route overlay model 1200 can include alocation-specific model 1204 superimposed over a map and a visual routeoverlay 1208 descriptive of a suggested navigational route to reach alocation.

A route overlay model 1200 can utilize the location-specific model 1204,a map database, and a determined navigational route to provide a userwith an enhanced map application experience. The route overlay model caninvolve the generation of a location-specific model 1204, the retrievalof a user computing device location, the retrieval of a locationaddress, and the retrieval of a map. The retrieved user location and theretrieved location address along with the retrieved map can be used todetermine a route to get from the user location to the address. Thedetermined route can be used to generate a visual route overlay 1208descriptive of a navigational route. The map, the location-specificmodel 1204, and the visual route overlay 1208 can be used to generate anaugmented map that depicts the location-specific model 1204 and thevisual route overlay 1208 superimposed over the map to providenavigational directions to a user with the added visual nuances of thethree-dimensional model.

In this example implementation, the route overlay model 1200 can be partof a map application that can further include various features such asroute previews 1202, ordered directions list, location labels (e.g.,transit location, restaurants, lodging, etc.), location markers (e.g.,waypoint, final destination, etc.), etc. The depicted route overlaymodel 1200 includes the location-specific model (i.e., a transitstation) superimposed on a map with a visual route overlay 1208providing directions to the location marker 1206. The route overlaymodel can be provided via a user interface, which can include atwo-dimensional map and directions 1210 along with the three-dimensionalrepresentation.

Example Methods

FIG. 6 depicts a flow chart diagram of an example method to performaccording to example embodiments of the present disclosure. AlthoughFIG. 6 depicts steps performed in a particular order for purposes ofillustration and discussion, the methods of the present disclosure arenot limited to the particularly illustrated order or arrangement. Thevarious steps of the method 600 can be omitted, rearranged, combined,and/or adapted in various ways without deviating from the scope of thepresent disclosure.

At 602, a computing system can obtain a location query from a usercomputing device. The location query can be a search query input into auser interface by a user.

At 604, the computing system can process the location query to determinea location.

At 606, the computing system can access a three-dimensional assetdatabase to retrieve a three-dimensional model associated with thelocation. The three-dimensional model can include one or morearchitectural structures and may include the determined location.

At 608, the computing system can generate a location-specific model bysegmenting the three-dimensional model to isolate the location fromother objects in the three-dimensional model. The segmenting can bethree-dimensional segmentation or image segmentation to remove trees,roads, and other buildings from the three-dimensional model to generatethe location-specific model. The segmentation can involve utilizing aplurality of images of the location to segment the three-dimensionalmodel, and the plurality of images may include one or more aerialimages. In some implementations, the location-specific model can includea model of an architectural structure (e.g., landmark, monument,historical building, etc.).

At 610, the computing system can provide the location-specific model tothe user computing device. The location-specific model can be providedas part of an interactive environment, a virtual-reality experience, anaugmented-reality experience, and/or a pre-rendered media item. In someimplementations, the location-specific model can be provided withlocation information, which can include a location name, a locationaddress, location images, and/or location facts.

FIG. 7 depicts a flow chart diagram of an example method to performaccording to example embodiments of the present disclosure. AlthoughFIG. 7 depicts steps performed in a particular order for purposes ofillustration and discussion, the methods of the present disclosure arenot limited to the particularly illustrated order or arrangement. Thevarious steps of the method 700 can be omitted, rearranged, combined,and/or adapted in various ways without deviating from the scope of thepresent disclosure.

At 702, a computing system can obtain a location query. The locationquery may include one or more search terms and/or one or more images.

At 704, the computing system can process the location query to determinea location. The location can be determined using a search engine,location tags, and or other determination methods.

At 706, the computing system can access a three-dimensional assetdatabase to retrieve a three-dimensional model and generate alocation-specific segment. The three-dimensional asset database canleverage a large database of images to generate a plurality ofthree-dimensional models that can be associated with locations and madereadily searchable. The generation of the location-specific segment caninclude removing objects from the three-dimensional model that are notpart of the specific determined location. In some implementations, thelocation-specific segment can be generated by a segmentation processthat isolates a location-specific model by removing portions of thethree-dimensional model.

At 708, the computing system can obtain situation data. The situationdata can be descriptive of a time specific situation associated with thedetermined location. The time specific situation can be a weather state,a traffic level, a crowd size, and/or train locations.

At 710, the computing system can process the situation data to determinea respective simulation asset. The respective simulation asset can be aset of data that can enable the rendering of the specific situation intoa simulated event.

At 712, the computing system can retrieve the respective simulationasset. The simulation asset can be retrieved from a database ofsimulation assets that store a plurality of situation specificsimulation assets.

At 714, the computing system can generate a simulation event. Thesimulation event can include the location and the time specificsituation based on the location-specific segment and the respectivesimulation asset. For example, the location can be Golden Gate Bridge,and the time specific situation can be rain with heavy traffic. Thelocation-specific segment can be a three-dimensional model of the GoldenGate Bridge, and the simulated event can be the Golden Gate Bridge modelwith heavy traffic and rain rendered into the model.

FIG. 8 depicts a flow chart diagram of an example method to performaccording to example embodiments of the present disclosure. AlthoughFIG. 8 depicts steps performed in a particular order for purposes ofillustration and discussion, the methods of the present disclosure arenot limited to the particularly illustrated order or arrangement. Thevarious steps of the method 800 can be omitted, rearranged, combined,and/or adapted in various ways without deviating from the scope of thepresent disclosure.

At 802, a computing system can reconstruct a three-dimensionalenvironment to generate a three-dimensional mesh and a texture. Thereconstruction can include a database of collected map data, image data,and other location data.

At 804, the computing system can obtain a plurality of images of theenvironment.

At 806, the computing system can process the plurality of images with amachine-learned model to segment the plurality of images into parts. Thesegmentation can be used to understand the meshes and textures to findthe underlying semantic structures of the environment. The segmentationcan utilize semantic manifolds for the image segmentation. A graph cutalgorithm can be applied to truly segment the mesh into parts.

At 808, the computing system can project the image parts into thethree-dimensional mesh.

At 810, the computing system can remove one or more objects from thereconstructed environment to generate an augmented environment. The oneor more objects can include other architectural structures, trees,sidewalks, and other obstructing objects. The removal can be completedautomatically or can include obtaining user inputs to remove one or morefeatures in the mesh.

At 812, the computing system can store the augmented environment. Theaugmented environment can be compressed before being stored and may bestored in a three-dimensional asset database.

Additional Disclosure

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

What is claimed is:
 1. A computer-implemented method for supplyingthree-dimensional modeling to a user, the method comprising: obtaining,by a computing system comprising one or more computing devices, alocation query associated with a user computing device; processing, bythe computing system, the location query to determine a location;accessing, by the computing system, a three-dimensional asset databaseto retrieve a three-dimensional model that models the location incombination with one or more other, different locations; segmenting, bythe computing system, the three-dimensional model to generate alocation-specific segment of the three-dimensional model thatcorresponds to the location, wherein the location-specific segmentmodels the location in isolation from the one or more other, differentlocations; and providing, by the computing system, the location-specificsegment of the three-dimensional model or one or more renderings of thelocation-specific segment of the three-dimensional model to the usercomputing device.
 2. The method of claim 1, wherein thelocation-specific segment of the three-dimensional model is generated byimage segmentation using a plurality of images associated with thelocation.
 3. The method of claim 1, wherein the location-specificsegment of the three-dimensional model is generated based at least inpart on sensor data descriptive of the location.
 4. The method of claim1, wherein the location-specific segment of the three-dimensional modelcomprises an architectural structure or a landmark.
 5. The method ofclaim 1, wherein the location-specific segment of the three-dimensionalmodel is an augmented-reality asset.
 6. The method of claim 1, furthercomprising: providing a virtual-reality experience comprising thelocation-specific segment of the three-dimensional model.
 7. The methodof claim 1, wherein the location-specific segment of thethree-dimensional model is provided in an interactive environment. 8.The method of claim 7, wherein the interactive environment comprises theone or more renderings of the location-specific segment of thethree-dimensional model in combination with one or more two-dimensionalmaps.
 9. The method of claim 7, wherein the interactive environmentcomprises a portion of a search results webpage that includes additionalsearch results responsive to the location query.
 10. The method of claim1, wherein the location-specific segment of the three-dimensional modelis provided in a pre-rendered video.
 11. The method of claim 1, furthercomprising: retrieving, by the computing system, an address of thelocation; and providing, by the computing system, the address of thelocation with the location-specific segment of the three-dimensionalmodel or the one or more renderings of the location-specific segment ofthe three-dimensional model.
 12. The method of claim 1, furthercomprising: obtaining, by the computing system, situation data, whereinthe situation data is descriptive of a time-specific situationassociated with the location; processing, by the computing system, thesituation data to determine a respective simulation asset; retrieving,by the computing system, the respective simulation asset; andgenerating, by the computing system, a simulated event comprising thelocation and the time-specific situation based on the location-specificsegment of the three-dimensional model and the respective simulationasset.
 13. The method of claim 12, wherein the situation data comprisestraffic data; and wherein the respective simulation asset comprisessimulated traffic.
 14. The method of claim 12, wherein the situationdata comprises crowd data; and wherein the respective simulation assetcomprises a simulated crowd of a size based on the crowd data.
 15. Themethod of claim 12, wherein the situation data comprises weather data;and wherein the respective simulation asset comprises a simulatedweather state.
 16. A computing system, the computing system comprising:one or more processors; one or more non-transitory computer readablemedia that collectively store instructions that, when executed by theone or more processors, cause the computing system to performoperations, the operations comprising: obtaining a location query from auser computing device; processing the location query to determine alocation; accessing a three-dimensional asset database to retrieve athree-dimensional model associated with the location; generating alocation-specific model by segmenting the three-dimensional model toisolate the location from other objects in the three-dimensional model;and providing the location-specific model to the user computing device.17. The computing system of claim 16, wherein the operations furthercomprise: generating a map comprising the location-specific model; andproviding the map to the user computing device.
 18. The computing systemof claim 17, wherein the operations further comprise: obtaining locationdata for the user computing device, wherein the location data isdescriptive of a user location; obtaining an address for the locationassociated with the location-specific model; determining a route for auser to get to the address from the user location; generating a visualroute overlay based on the route; generating an augmented map, whereinthe augmented map comprises the visual route overlay and the map; andproviding the augmented map to a user computing device.
 19. One or morenon-transitory computer readable media that collectively storeinstructions that, when executed by one or more processors, cause acomputing system to perform operations, the operations comprising:obtaining a user input that specifies a location query that relates to alocation; providing the location query to a server system over anetwork; receiving, from the server system over the network, alocation-specific segment of a three-dimensional model or one or morerenderings of the location-specific segment of a three-dimensionalmodel, wherein the three-dimensional model models the location incombination with one or more other, different locations, and wherein thelocation-specific segment of a three-dimensional model has beengenerated by segmenting the three-dimensional model to isolate thelocation from the one or more other, different locations; and providingfor display at least a first rendering of the location-specific segmentof the three-dimensional model.
 20. The one or more non-transitorycomputer readable media of claim 19, wherein providing for display atleast the first rendering of the location-specific segment of thethree-dimensional model comprises providing for display at least thefirst rendering in a search results webpage.